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1 TITLE OF THE INVENTION 

2 DISTANCE CORRECTING APPARATUS OF SURROUNDINGS MONITORING SYSTEM 

3 AND VANISHING POINT CORRECTING APPARATUS THEREOF. 
4 

5 BACKGROUND OF THE INVENTION 

6 1. Field of the invention 

7 The present invention relates to a distance correcting 

8 apparatus of a surroundings monitoring system for correcting 

9 distance information containing errors caused by a positional 

10 deviation of a stereoscopic camera and to a vanishing point 

11 correcting apparatus of the system. 

12 2. Discussion of the background art 

13 In recent years, a stereoscopic surrounding 

14 monitoring apparatus using a pair of left and right cameras, that 

15 is, a stereoscopic camera, having solid image element like CCD 

16 mounted on a vehicle and the like has been watched by concerned 

17 engineers. To detect a distance to an object, first respective 

18 pixel blocks having coincidence of brightness are found in left 

19 and right images (stereo matching) , then distance data are 

20 calculated according to the principle of triangulation from a 

21 parallax, namely a relative deviation amount, between both pixel 

22 blocks. Consequently, in order to calculate distance data with 

23 high reliability, it is desirable that there exists no positional 

24 deviation 'other than the parallax in a pair "of left and right 

25 images (stereo images) . In actual world, however, the 
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1 stereoscopic camera has some amount of positional errors such 

2 as horizontal or vertical deviations (parallel deviations) , a 

3 rotational deviation and the like, caused when the camera is 

4 installed on a vehicle and the like. Particularly, the horizontal 

5 deviation directly produces an error in an parallax and as a result 

6 the distance calculated based on the parallax differs from a real 

7 one . 

8 With respect to this, Japanese Patent Application 

9 Laid-open No. Toku-Kai-Hei 10-307352 discloses a technology in 

10 which the positional deviation of the stereoscopic camera is 

11 corrected by applying a geometric transformation to the 

12 stereoscopic image. That is, when an initial adjustment of the 

13 positional deviation is made or when a readustment of the 

14 positional deviation generated by aged deterioration is made, 

15 a dedicated correction detecting device is connected with an image 

16 correction apparatus performing the affine transformation to 

17 calculate the difference of angle of view, a rotational deviation 

18 or a parallel deviation of the stereoscopic image obtained by 

19 imaging a specified pattern for adjustment and to establish 

20 (reestablish) parameters of the affine transf omation according 

21 to the result of the calculation. The positional deviation is 

22 eguivalently corrected by applying the affine transformation 

23 to images based on thus established affine parameters. 

24 However, according to the aforesaid prior art, a 

25 special adjustment pattern is imaged by the stereoscopic camera 
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1 and the deviation is corrected based on the position of the pattern 

2 in images. Accordingly, when the correction is performed, it is 

3 necessary to interrupt the ordinary surroundings monitoring 

4 control and as a result this prior art is not suitable for a real 

5 time processing in which the monitoring control is carried out 

6 concurrently. 
7 

8 SUMMARY OF THE INVENTION 

9 It is an object of the present invention to provided 

10 a surroundings monitoring apparatus capable of correcting a 

11 parallax including errors, in particular, an error caused by 

12 horizontal deviation, in parallel with a surroundings monitoring 

13 control. It is further object of the present invention to provide 

14 a surroundings monitoring apparatus in which the accuracy of 

15 measuring distance is raised by using the corrected parallax. 

16 It is another object of the present invention to provide a 

17 surroundings monitoring apparatus in which, when three- 

18 dimensional information of an object is obtained using a vanishing 

19 point established beforehand, the accuracy of three-dimensional 

20 information of the object is raised by correcting this vanishing 

21 point. 

22 To achieve these objects, a distance correcting 

23 apparatus of a surroundings monitoring system, comprises a stereo 

24 imaging means for s tereoscopically taking a 'pair of images, a 

25 parallax calculating means for calculating a parallax based on 



3 



1 the pair of images, a distance calculating means for calculating 

2 a distance to an object based on the parallax and a parameter 

3 for correcting the distance, an approximation line calculating 

4 means for calculating a plurality of approximation lines 



5 extending in the distance direction in parallel with each other 

6 based on the images, a vanishing point calculating means for 

7 calculating a vanishing point of the images from a point of 

8 intersection of the approximation lines and a parameter 

9 correcting means for correcting the parameter based on the 



10 vanishing point. 
11 

12 BRIEF DESCRIPTION OF THE DRAWINGS 

13 Fig. 1 is a block diagram showing a construction of 

14 a stereoscopic type vehicle surroundings monitoring apparatus 

15 according to a first embodiment of the present invention; 

16 Fig. 2 is a flowchart showing steps for correcting a 

17 parallax according to a fist embodiment; 

18 Fig. 3 is a flowchart continued from Fig. 2; 

19 Fig. 4 is a flowchart showing steps for updating a 

20 parallax correction value DP according to a first embodiment; 

21 Fig. 5 is a flowchart showing steps for updating a 

22 parallax correction value DP according to a second embodiment; 

23 Fig. 6 is a block diagram showing a construction of 

24 a stereoscopic type vehicle surroundings monitoring apparatus 

25 according to a third embodiment of the present invention; 
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1 Fig. 7 is a flowchart showing steps for updating a 

2 parallax correction value SHFT1; 

3 Fig. 8 is a diagram for explaining a calculated road 

4 height; 

5 Fig. 9 is a diagram showing a relationship between a 

6 calculated road height and an actual road height; 

7 Fig. 10 is a diagram for explaining a deviation caused 

8 by the difference between an actual road height and a calculated 

9 road height; 

10 Fig. 11 is a diagram showing an example of a lane marker 

11 model; 

12 Fig. 12 is a diagram for explaining lane marker edges 

13 of a reference image; 

14 Fig. 13 is a diagram for explaining a calculation method 

15 of a vanishing point in a reference image; 

16 Fig. 14 is a block diagram showing a construction of 

17 a stereoscopic type vehicle surroundings monitoring apparatus 

18 according to a fourth embodiment of the present invention; 

19 Fig. 15 is a flowchart showing steps continued from 

20 Fig. 2 according to a fourth embodiment; 

21 Fig. 16 is a diagram showing an example of an image 

22 of an indoor robot; and 

23 Fig. 17 is a diagram showing an example of an image 

24 of a scenery in front of a railway rolling stock. 
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1 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

2 Fig. 1 is a block diagram of a stereoscopic type 

3 surroundings monitoring apparatus using an adjusting apparatus 

4 concerned with the embodiment. A stereoscopic camera for imaging 

5 a surrounding scenery of a vehicle is composed of a pair of 

6 cameras 1, 2 incorporating an image sensor such as CCD and the 

7 like and mounted in the vicinity of a room mirror of the vehicle. 

8 The cameras 1, 2 are mounted at a specified interval in the 

9 transversal direction of the vehicle. A main camera 1 is for 

10 obtaining a reference image data and is mounted on the right side 

11 when viewed in the traveling direction of the vehicle. On the 

12 other hand, a sub camera 2 is for obtaining a comparison image 

13 data and is mounted on the left side when viewed in the traveling 

14 direction of the vehicle. In a state of the cameras 1, 2 

15 synchronized with each other, analogue images outputted from the 

16 respective cameras 1, 2 are adjusted in an analogue interface 

17 3 so as to coincide with an input range of circuits at the latter 

18 stage. Further, the brightness balance of the images is adjusted 

19 in a gain control amplifier (GCA) 3a of the analogue interface 

20 3. 

21 The analogue image signals adjusted in the analogue 

22 interface 3 are converted into digital images having a specified 

23 number of brightness graduations {for example, a grayscale of 

24 256 graduations) by an A/D converter 4.' Respective data 

25 digitalized are subjected to an affine transformation in a 
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1 correction circuit 5. That is, the positional error of the 

2 stereoscopic cameras 1, 2 which is caused when the cameras 1, 

3 2 are installed, generates deviations in stereoscopic images such 

4 as a rotational deviation, parallel deviation and the like. The 

5 error is equivalently corrected by applying the affine 

6 transformation to the images. In this specification, a term 

7 "affine transformation" is used for comprehensively naming a 

8 geometrical coordinate transformation including rotation, 

9 movement, enlargement and reduction of images. The correction 

10 circuit 5 applies a linear transformation expressed in Formula 

11 1 to original images using four affine parameters 6 , K, SHFTI 

12 and SHFTJ. 

13 [Formula 1] 



17 where (i, j) is coordinates of an original image and (i' , j') 

18 is coordinates after transformation. Further, affine parameters 

19 SHFTI, SHFTJ mean a transference in a w i" direction (horizontal 

20 direction of image) , a transference in a " j" direction (vertical 

21 direction of image), respectively. Further, affine parameters 

22 0 , K indicate a rotation by 6 , an enlargement (reduction in case 

23 of |K|< 1) by K times, respectively. The affine transformation 

24 applied to the stereoscopic image assures a coincidence of the 

25 horizontal line in both images, which is essential for securing 



14 



15 




16 



7 



1 the accuracy of the stereo matching. The hardware constitution 

2 of the correction circuit 5 is described in Japanese Patent 

3 Application Laid-open No. Toku-Kai-Hei 10-307352. If necessary, 

4 the reference should be made to the disclosure. 

5 Thus, through such image processing, the reference 

6 image data composed of 512 pixels horizontally and 200 pixels 

7 vertically are formed from output signals of the main camera 1. 

8 Further, the comparison image data having the same vertical length 

9 as the reference image and a larger horizontal length than the 

10 reference image, for example composed of 640 pixels horizontally 

11 and 200 pixels vertically, are formed from output signals of the 

12 sub camera 2. The coordinate system i-j of image on a two- 

13 dimensional plane has an origin at the left below corner of the 

14 image, an i coordinate in the horizontal direction and a j 

15 coordinate in the vertical direction. One unit of the coordinate 

16 system is one pixel. These reference image data and comparison 

17 image data are stored in an image data memory 7. 

18 A stereo calculating circuit 6 calculates a parallax 

19 d based on the reference image data and the comparison image data. 

20 Since one parallax d is produced from one pixel block constituted 

21 by 4 x 4 pixels, 128 x 50 parallax data are calculated per one 

22 reference image of a frame size. In calculating a parallax di 

23 of a given pixel block in a reference image, first a corresponding 

24 pixel block of a comparison image is identified by searching an 

25 area having the same brightness as that given pixel block of the 

8 



1 reference image. As well known, the distance from the camera to 

2 an object projected in a stereo image is expressed as a parallax 

3 in the stereo image, namely a horizontal deviation amount between 

4 the reference and comparison images. Accordingly, in searching 

5 the comparison image, the search is performed on the same 

6 horizontal line (epipolar line) as a j coordinate of the reference 

7 image. In the stereo calculating circuit 6, the correlation is 

8 evaluated for every pixel block between the object pixel block 

9 and the searching pixel block while shifting a pixel one by one 

10 on the epipolar line (stereo matching). 

11 The correlation between two pixel blocks can be 

12 evaluated for example using a city block distance which is one 

13 of well known evaluation methods. The stereo calculating circuit 

14 6 obtains a city block distance for every area (having the same 

15 area size as the object pixel block) existing on an epi-polar 

16 line and identifies an area whose city block distance is minimum 

17 as a correlation object of the object pixel block. The deviation 

18 amount between the object pixel block and the identified 

19 correlation object equals to a parallax di . The hardware 

20 constitution for calculating the city block distance and the 

21 method of determining the correlation object is disclosed in 

22 Japanese Patent Application No. Toku-Kai-Hei 5-114009. If 

23 necessary, the reference should be made to the disclosure. The 

24 parallax d calculated by the stereo calculating circuit 6 is 

25 stored in the distance data memory 8 . 
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1 The micro-computer 9 or when seeing it from a functional 

2 point of view, a recognition section 10 which is a functional 

3 block, read image data of a reference image out from an image 

4 data memory 7 and recognizes an object (for example, a preceding 

5 vehicle and the like) projected in the reference image using a 

6 known image recognition technique. Further, the recognition 

7 section 10 calculates a distance Z to the object according to 

8 the following formula parameterizing a parallax d read out from 

9 the distance data memory 8. 

10 [Formula 2] 

11 Z = KZH / (d - DP) 

12 where KZH is a constant (base line length of camera / horizontal 

13 angle of view) and DP is a vanishing point parallax. In this 

14 embodiment, the vanishing point parallax DP is a parallax 

15 correction value (variable) which is calculated in a correction 

16 calculating section 13. 

17 Further, the recognition section 10 performs a 

18 recognition of road configurations. Road configurations, that 

19 is, left and right lane markers (passing line, no passing line 

20 and the like) are expressed in a three-dimensional space as 

21 functions having parameters established so as to coincide with 

22 actual road configurations such as straight roads, curved roads 

23 or up-and-down roads. In this embodiment, a term "lane marker" 

24 represents a continuous white line-like marker drawn on a road, 

25 although the present invention is not limited to such lane markers . 
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1 The method of calculating a lane marker model according to this 

2 embodiment will be described by reference to Fig. 12. 

3 First, a white line edge Pedge, namely, a portion 

4 showing a large variation in brightness, is identified. The white 

5 line edge Pedge is searched separately for the left side and right 

6 side of a lane, respectively. A plurality of left white line edges 

7 Pedgel and a plurality of right white line edges Pedge2 are 

8 identified respectively. Specifically, the brightness edges 

9 satisfying following three conditions are recognized as white 

10 line edges Pedge. 

11 (Conditions of white line edge) 

12 1 . Brightness variation is larger than a specified value 

13 and pixels on the outer side (edge side of image) have a larger 

1 4 brightness than those on the inner side (central side of image) . 

15 The white line edges Pedge caused by the left and right 

16 lane markers are brightness edges at the boarder of lane marker 

17 and paved surface, as shown in Fig. 12. 

18 2 . With respect to candidates of the white line edge Pedge 

19 aa.tisfying the condition 1, another edge exists outside of one 
2 0 edge on the same horizontal line as the candidates and brightness 

21 of pixels on the inner side is larger than that of pixels on the 

22 outer side. 

23 Since the lane marker has a specified width, there is 

24 another boarder on the outer side of the white line edge Pedge. 

25 This condition is provided in view of the feature of lane marker. 
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1 3 • With respect to pixel blocks .including the white line 

2 edge Pedqe satisf yin g the condi tio n \. a parallax d has been 

3 calculated. 

4 If there is no parallax d where a white line edge exists, 

5 the white line edge Pedge is not effective for recognizing a road 

6 configuration. 

7 The recognition section 10 calculates coordinates (X, 

8 Y, Z) in real space by substituting coordinates (i, j) and its 

9 parallax d for every identified white line edge Pedge into the 

10 following Formula 3 and Formula 4. 

11 [Formula 3] 

12 Y = CAH - Z(JV -j)PWV 
13 

14 [Formula 4] 

15 X = r/2 + Z(IV -i)PWH 

16 where CAH is an installation height of cameras 1, 2; r is an 

17 interval between cameras 1, 2; PWV and PWH are a vertical and 

18 horizontal angle of view per one pixel, respectively; IV and JV 

19 are an i coordinate and j coordinate of a vanishing point V 

20 established, respectively. 

21 Further, the coordinate system in real space comprises 

22 an origin placed on the road surface immediately beneath of the 

23 center of the cameras 1, 2, X axis extending in the widthwise 

24 direction of the vehicle, Y axis extending in the vertical 

25 direction of the vehicle and Z axis extending in the longitudinal 
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1 direction of the vehicle. When the coordinates (i, j) and the 

2 parallax d of an object (a preceding vehicle, a solid object, 

3 a road and the like) projected on the image are identified, the 

4 coordinates (X, Y, Z) of the object in real space can be 

5 unconditionally identified according to the transformation 

6 formulas shown in Formulas 2 through 4. 

7 A lane marker model is identified based on the 

8 coordinates (X, Y, Z) of thus identified respective white line 

9 edges Pedge in real space. The lane marker model is prepared in 

10 such a manner that approximation lines are obtained for every 

11 specified interval with respect to each of the left and right 

12 white line edges Pedgel, Pedge2 within a recognition range (for 

13 example, a range of 84 meters away in front of the vehicle from 

14 camera) and thus obtained approximation lines are combined like 

15 broken lines. Fig. 11 shows an example of a lane marker model 

16 in which the recognition range is divided into seven segments 

17 and the left and right white line edges Pedgel, Pedge2 for each 

18 segment are approximated to a linear equation expressed as follows 

19 according to the least square method. 

20 [Formula 5] 

21 (Left lane marker model L) 

22 X = a L -Z + b L 

23 Y = c L -Z + d L 

24 (Right lane marker model R) 

25 X = a R -Z + b R 
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1 Y = c R -Z + d R 

2 These lane marker models L, R are constituted by a curve 

3 function (X = f(Z)) for expressing a curvature of road and a 

4 gradient function (Y = f(Z)) for expressing a gradient or 

5 condition of unevenness of road. Accordingly, the three- 

6 dimensional feature of the road in real space can be expressed 

7 by the left and right lane marker models L, R. Respective white 

8 line edges and left and right lane marker models L, R calculated 

9 in the recognition section 10 are transmitted to a correction 

10 calculating section 13. 

11 The recognition section 10 actuates a warning device 

12 11 such as a display monitor or a speaker when it is judged that 

13 a warning is needed based on the result of recognition of preceding 

14 vehicles or road configurations. Further, the recognition 

15 section 10 controls a control device 12 to carry out miscellaneous 

16 vehicle controls such as engine output control, shift control 

17 of automatic transmission, brake control and the like. 

18 Next, the method of correcting distance information 

19 according to the embodiment will be briefly described by reference 

20 to Fig. 8. 

21 Assuming that the Z axis of the vehicle is always 

22 horizontal with respect to an even road without up-and down, that 

23 is, there is no pitching of the vehicle, the height Y of the road 

24 surface is expressed by a line Lr with a gradient a (a =0) . This 

25 line Lr is called an actual road surface height. Letting 
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1 coordinates of a point pi (hereinafter referred to as a road 

2 surface point) projected on the reference image be (il, jl) and 

3 letting its parallax be dl, the position of this road surface 

4 point pi in real space is identified unconditionally as 

5 coordinates (xl, yl, zl) . 

6 [Formula 6] 

7 zl = KZH/ (dl - DP) 
8 

9 [Formula 7] 
10 yl = CAH - zl(JV - jl)PWV 

11 

12 [Formula 8] 

13 xl = r/2 + zl(IV - il)PWH 
14 

15 In case where a flat road without up-and-down horizontally exists, 

16 if the distance zl calculated from the parallax dl includes no 

17 error, the height yl calculated from Formula 7 should be 0. That 

18 is, if the value of the distance zl is identical to an actually 

19 measured value, a line Lr' (hereinafter, referred to as a 

20 calculated road surface height) connecting an origin and the road 

21 surface point pi agrees with the actual road surface height. 

22 Namely, the gradient of the calculated road surface height Lr' 

23 becomes 0. On the other hand, in case where the value of the 

24 distance zl contains errors and differs from the actually measure 

25 value, the height yl calculated from Formula 7 is not equal to 
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1 0, the calculated road height Lr' having a specified gradient 

2 a' (a' = yl/zl ^0). 

3 The reason why the calculated height yl is not equal 

4 to 0 is that the parallax dl containing errors due to the effect 

5 of the horizontal deviation of the stereoscopic camera is 

6 calculated and these errors are not properly offset by the 

7 vanishing point parallax DP (corresponding to a parallax 

8 correction value) . Hence, if a deviation amount of the gradient 

9 a' (a' ^ 0 ) of the calculated road surf ace height Lr' with respect 

10 to the gradient a of the actual road surface height Lr is known, 

11 a deviation amount A DP between the proper value of the vanishing 

12 point parallax DP and the current value can be calculated. 

13 First, in case where the vanishing point parallax DP 

14 is an optimum vale enough to be able to completely offset the 

15 errors, the gradient value of the calculated road surface height 

16 Lr' (agrees with the the gradient of the actual road surf ace height 

17 Lr ) is a . Accordingly, the gradient a is expressed based on Formula 

18 6 and Formula 7 which have been described as follows: 

19 [Formula 9] 

20 a =*L, 

zl 

21 CAH 

= KZH (dl ~ DP) ~~ (JV ~ jl) PWV 

22 

23 On the other hand, in case where the vanishing point 

24 parallax is a value DP' which deviates from the proper value DP, 

25 the gradient a' of the calculated road surface height Lr' is 
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1 expressed in the following formula: 

2 [Formula 10] 



zl 

CAH 

= — -(dl-DPO-(JV-jl)PWV 



- (DP' - DP) 
KZH v 



6 Eliminating d, j based on the formulas 9 and 10, 

7 following formula is obtained: 

8 [Formula 11] 

9 CAH 
a 

10 

11 Transforming the formula 11 to obtain DP - DP' , 

12 that is, the deviation amount ADP of the vanishing point parallax: 

13 [Formula 12] 
14 

4DP = DP-DP' 
KZH 

16 = CAH (a '- a) 

17 The gradient a of the actual road height Lr is 0. On 

18 the other hand, the gradient a' of the calculated road height 

19 Lr' can be identified based on the parameter c of the lane marker 

20 model L, R (Y = c. Z + d) calculated in the recognition section. 

21 Similarly to the gradient a' of the calculated road surface height 

22 Lr' , when the horizontal deviation of the stereoscopic camera 

23 exists, the error caused by the deviation effects on the lane 

24 marker model L, R. Hence, letting the mean value of parameters 

25 cL, cR of the left and right lane marker model L, R up to a 
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predetermined distance (for example a range from 0 to Z2) be C, 



2 it is possible to regard this value C as a gradient a' of the 



3 calculated road surface height Lr' . Further, substituting a = 

4 0, a' = C into the formula 12, the deviation amount A DP of the 

5 vanishing point parallax is expressed by the following formula 

6 finally: 

7 [Formula 13] 

9 CAH 

10 As se en from the formula 13, the result of multiplying 

11 the parameter C by a constant (KZH/CAH) is the deviation amount 

12 A DP of the vanishing point parallax. Hence, by adding the 

13 deviation amount A DP to the vanishing point parallax DP, the 

14 calculated road surface height Lr' can be made identical to the 

15 actual road height Lr (a' = a = 0 . That is, the error of the parallax 

16 d caused by the horizontal deviation of the stereoscopic camera 

17 can be eliminated by using the vanishing point parallax DP 

18 properly established based on the deviation amount ADP calculated 

19 according to the formula 13. As a result, even in a case where 

20 a horizontal deviation of the stereoscopic camera exists, an 

21 accurate distance Z can be calculated by properly establishing 

22 the vanishing point parallax DP which is a parallax correction 

23 value. 

24 The description above is based on a premise that the 

25 flat road without up-and-down is always horizontal with respect 
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1 to Z-axis. However, in practice, an actual road surface height 

2 L of the flat road does not always agree with Z-axis due to the 

3 affect of the pitching motion of the own vehicle. For example, 

4 when the own vehicle directs upward (sky side), the gradient a 

5 of the actual road surface height Lr becomes a negative value 

6 and when the own vehicle directs downward (ground side) , the 

7 gradient a of the actual road surface height Lr becomes a positive 

8 value. When the gradient a of the actual road height Lr is rendered 

9 to be 0 as mentioned before, the deviation amount ADP itself has 

10 an error due to the effect of pitching. From the view point of 

11 improving the accuracy of a calculated distance, it is necessary 

12 to properly calculate the gradient a of the actual road surface 

13 height Lr. 

14 " A vanishing point" is identified based on a two- 

15 dimensional (i-j plane) positional information of the left and 

16 right lane markers in the reference image and then a gradient 

17 a of the actual road surface height Lr is calculated from this 

18 "vanishing point". Here, the term "vanishing point" is defined 

19 to be an infinitely far point (infinite point) , that is, a point 

20 where all parallel lines extending in the depth (distance) 

21 direction converge at the infinite far image. For example, when 

22 a rectangular parallelepiped disposed in a three-dimensional 

23 space is mapped through a camera on a two-dimensional plane, the 

24 parallel lines constituting the rectangular parallelepiped 

25 always meet together at a point. This point of intersection is 
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1 "a vanishing point". In the vehicle surroundings monitoring 

2 apparatus for imaging the frontal scene , this example corresponds 

3 to a case where the left and right lane markers on respective 

4 road sides run ahead in parallel with each other in the depth 

5 (distance) direction of the image. Since the left and right lane 

6 markers are in parallel with each other, the left and right lane 

7 markers in the picture image are approximated to straight lines 

8 respectively, letting the intersection of these lines be a 

9 vanishing point V2d (IV2D, JV2D) . 

10 Specifically, as shown in Fig. 13, a plurality of left 

11 white line edges Pedgl are approximated to a straight line to 

12 obtain an approximation line LI and similarly a plurality of right 

13 white line edges Pedg2 are approximated to a straight line to 

14 obtain an approximation line L2 . In order to raise the accuracy 

15 in calculating the vanishing point JV2D, it is preferable that 

16 only the white line edges within a specified range of distance 

17 (for example, 0 to Z2) are used for calculating the approximation 

18 line. The range of distance, if it is too short, the accuracy 

19 of the approximation lines LI, L2 and if it is too long, the amount 

20 of calculations increases or there is a decreasing chance of the 

21 lane marker projected on the line, that is, it is difficult to 

22 create the condition of lane marker suitable for calculating the 

23 vanishing point JV2D. The intersection of these approximation 

24 lines LI, L2 is a vanishing point V2d. The gradient a of the actual 

25 road surface height Lr can be identified if the j -coordinate JV2D 



20 



1 is known. Accordingly, in the description hereinafter, the 

2 j-coordinate JV2D of the vanishing point V2d is referred to as 

3 "actual vanishing point" for the purpose of discriminating from 

4 the established vanishing point JV. 

5 Fig. 9 is a diagram showing the relationship between 

6 the actual road surface height Lr and the calculated road surface 

7 height Lr' . The stereoscopic camera is mounted on the vehicle 

8 in such a manner that the vanishing line Lv connecting the 

9 installation height CAH of the camera and the actual vanishing 

10 point JV2D is in parallel with the actual road surface height 

11 Lr. In case where the own vehicle generates pitchings, the 

12 gradient of the actual road surface height Lr varies and at the 

13 same time the gradient of the vanishing line Lv also varies. That 

14 is, regardless of the existence or nonexistence of the pitching 

15 of the own vehicle, the gradient of the actual road surface height 

16 Lr always agrees with that of the vanishing line Lv (both gradients 

17 are a) . That is to say, even in case where the vehicle has a pitching 

18 motion, the vanishing line Lv is always in parallel with the actual 

19 road surface height Lr . Consequently, the gradient of the actual 

20 road surface height Lr can be identified by obtaining the gradient 

21 a of the vanishing line Lv. If this gradient a is known, the 

22 vanishing point parallax DP can be calculated by transforming 

23 the formula as follows. 

24 First, after substituting the vanishing point JV2D 

25 into a variable j of the formula 3, obtaining the gradient (Y/Z) 
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1 on Z-Y plane: 

2 [Formula 14] 

3 a = ( JV2D - JV) PWV 

4 As seen from the formula, if the actual vanishing point 

5 JV2D is identified, the gradient a (corresponding to the gradient 

6 of the actual road surface Lr) height of the vanishing line Lv 

7 is identified unconditionally. 

8 Substituting the formula 14 into the formula 12, 

9 finally the following formula can be obtained: 

10 [Formula 15] 

11 KZH KZH 

4 DP = C (JV2D - JV)PWV 

12 CAH CAH v 

13 The formula 15 is obtained by subtracting a portion 

14 affected by the pitching as a correction term from the formula 

15 13. The correction term is obtained by multiplying the product 

16 of substituting the established vanishing point JV from the actual 

17 vanishing point JV2D by a predetermined constant KZH/CAH. 

18 Accordingly, if the current value of the vanishing point parallax 

19 DP is added by the deviation amount A DP, regardless of the 

20 existence or nonexistence of pitching of the own vehicle, the 

21 gradient a' of the calculated road surface height Lr' always 

22 agrees with the gradient a of the actual road surface height Lr. 

23 This means that the error caused by the horizontal deviation of 

24 the stereoscopic camera is offset by the vanishing point parallax 

25 DP and the distance Z is calculated as being actually measured. 
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1 The effect of pitching of the own vehicle exerts not only on the 

2 gradient a of the vanishing line Lv (and the actual road surface 

3 height Lr) but also on the gradient a' of the calculated road 

4 surface height Lr' . However, the deviation amount A DP is 

5 calculated such that the effect of pitching with respect to the 

6 gradient a and the effect of pitching with respect to the gradient 

7 a' are mutually offset (refer to the formula 12). Accordingly, 

8 an accurate deviation amountADP 

9 can be calculated without being affected by pitching of the 

10 vehicle. 

11 Next, the detailed description of the parallax 

12 correction according to this embodiment will be made by reference 

13 to flowcharts shown in Fig. 2 and Fig. 3. 

14 The correction calculating section 13 updates the 

15 value of the vanishing point parallax DP according to a series 

16 of steps and this value is fed back to the recognition section 

17 10. The flowcharts are executed repeatedly per cycle. 

18 First, at a step 1, the correction calculating section 

19 13 reads white line edges Pedge and lane marker models L, R 

20 calculated in the recognition section 10 of a reference image. 

21 Next, at steps 2 through 6, it is evaluated whether or not the 

22 reference image is in a suitable condition for calculating the 

23 vanishing point JV2D. First, at a step 2, it is judged whether 

24 or not the left and right lane markers exist' in the reference 

25 image which is an object of calculating the vanishing point JV2D. 
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1 That is, this can be judged by investigating whether or not the 

2 left and right lane marker models L, R have been calculated in 

3 the recognition section 10. Further, this may be judged by 

4 investigating whether or not the left white line edges Pedgel 

5 and the right white line edges Pedge2 have been calculated. At 

6 the step 2, in case where the judgment is negative, that is, in 

7 case where the left and right lane markers exist nowhere, since 

8 mutually parallel lines have not extracted, the vanishing point 

9 JV2D can be calculated. Hence, in order to maintain the safety 

10 of the control, the program goes to RETURN without changing the 

11 current value of the vanishing point parallax DP and the execution 

12 of this flowchart in the present cycle finishes. On the other 

13 hand, at the step 2, in case where the judgment is positive, the 

14 program goes to a step 3. 

15 At the step 3, the reliability of the left and right 

16 lane markers are verified. Specifically, following two things 

17 are evaluated. 

18 1. In case where the difference between the position of 

19 the lane marker in the previous cycle and the position of the 

20 lane marker in the present cycle is greater than a specified value, 

21 it is judged that the lane marker has a low reliability. 

22 Specifically, in case where the position of the white line edge 

23 Pedge detected in the previous cycle largely deviates from the 

24 position of the white line edge Pedge detected in the present 

25 cycle, the lane marker is judged to have a low reliability. 
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12. It is verified how far the lane marker extends in the 

2 depth direction of an image. The lane marker has at least some 

3 extent of length. Accordingly, taking the shift of the lane 

4 marker between frames into consideration, in case where the lane 

5 marker does not extend longer than a specified length, it is judged 

6 that this lane marker has a low reliability. 

7 After that, at a step 4, it is judged whether or not 

8 the lane marker is reliable and only when it is judged to be 

9 reliable, the program goes to a step 5. On the other hand, when 

10 it is judged that the lane marker can not be relied, the program 

11 goes to RETURN without changing the value of the vanishing point 

12 parallax DP. 

13 At the step 5, the linearity of the lane marker is 

14 evaluated. In order to calculate an accurate vanishing point JV2D, 

15 it is necessary that the left and right lane markers extend in 

16 line. That is, it is impossible to calculate an accurate vanishing 

17 point JV2D from curved lane markers. Hence, only in case where 

18 it is judged at a step 6 that the lane marker is a straight line, 

19 The program goes to a step 7 and otherwise the program goes to 

20 RETURN without changing the value of the vanishing point parallax 

21 DP. 

22 The linearity of the lane marker can be evaluated for 

23 example based on a lane marker model (curve function X = f(Z)) 

24 calculated in the recognition section 10 . Describing by reference 

25 to Fig. 11, first a gradient Al (mean value of gradients a L , a R 
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1 of left and right lane markers L, R, respectively) of the curve 

2 function within a specified distance range (for example 0 to Z2) 

3 on Z-X plane, is calculated. The gradient Al is a mean value of 

4 a gradient al in the first segment and a gradient a2 in the second 

5 segment. Next, a gradient A2 of the curve function within a 

6 specified distance range located ahead (for example Z2 to Z4) 

7 is calculated. The gradient A2 is is a mean value of a gradient 

8 a3 in the third segment and a gradient a4 in the fourth segment. 

9 Then, a difference (absolute value) between the gradients Al and 

10 A2 is obtained. If the difference is smaller than a threshold 

11 value, it is judged that the lane marker is a straight line. 

12 Steps after the step 7 are related to an up-dating of 

13 the vanishing point parallax DP. First, at the step 7, an 

14 approximation line LI of a plurality of left white line edges 

15 Pedgel existing within a specified range (for example, 0 to Z2) 

16 is calculated according to the least square method (refer to Fig. 

17 13) . Similarly, an approximation line L2 of a plurality of left 

18 white line edges Pedge2 existing within that range is calculated 

19 according to the least square method. 

20 At a step 8 following the step 7, as shown in Fig. 13, 

21 an point of intersection of the approximation lines LI, L2 is 

22 determined to calculate the vanishing point JV2D of the reference 

23 image. Further, at a step 9, a gradient a of the vanishing line 

24 Lv is calculated by substituting the vanishing point JV2D 

25 calculated at the step 8 into the formula 14. As described above, 
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1 obtaining the gradient a of the vanishing point Lv just means 

2 calculating a gradient a of the actual road surface height. 

3 Next, at a step 10, a gradient a' of the calculated 

4 road surface height Lr' is calculated. As mentioned before, the 

5 gradient a' is a parameter C calculated from the left and right 

6 lane marker models L, R. 

7 At a step 11, the correction of parallax, namely, an 

8 up-dating of the vanishing point parallax DP is performed. Fig. 

9 4 is a flowchart showing steps for up-dating the vanishing point 

10 parallax DP. First, at a step 21, a deviation amount A DP is 

11 calculated by substituting the parameter C and the vanishing point 

12 JV2D into the formula 15. 

13 At a step following the step 21, in order to secure 

14 the safety of control, the up-dating process of the vanishing 

15 point parallax DP is performed using a proportional control. That 

16 is, the value of the vanishing point parallax DP is up-dated by 

17 adding an value the deviation amount A DP calculated at the step 

18 21 and multiplied by a proportional constant k(0<k<l) to the 

19 present value of the vanishing point parallax DP. Further, at 

20 a step 23, the up-dated vanishing point parallax DP is outputted 

21 to the recognition section 10 and the execution of this flowchart 

22 in the present cycle finishes. 

23 The aforesaid flowchart is carried out in consecutive 

24 cycles. Therefore, even if such a situation that the vanishing 

25 point parallax DP is out of a proper value, occurs, the vanishing 
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1 point parallax DP gradually comes close to a proper value by 

2 carrying the flowchart out repeatedly. Hence, since the error 

3 of the distance Z caused by the horizontal deviation of the 

4 stereoscopic camera is gradually offset, the gradient a' of the 

5 calculated road surface Lr' converges to the gradient a of the 

6 actual road surface height Lr. 

7 According to the steps described above, the 

8 optimization of the vanishing point parallax DP proceeds in 

9 parallel with the normal monitoring control and even in case where 

10 the horizontal deviation of the stereoscopic camera occurs, the 

11 distance can be always calculated accurately. Accordingly, even 

12 in case where the position of the stereoscopic camera is changed 

13 from the initial position by aged deterioration of the camera 

14 or shocks applied to thereto, highly reliable distance 

15 information can be obtained stably. The highly reliable distance 

16 information provides surroundings monitorings with a 

17 reliability. 

18 Further, the left and right lane markers existing on 

19 both sides of the road are used as mutually parallel lines 

20 extending in the depth direction and needed for the calculation 

21 of the vanishing point JV2D of the reference image. In this 

22 embodiment, it is j udged whether or not the lane marker is suitable 

23 for calculating the vanishing point JV2D by evaluating the 

24 linearity of the lane marker or the positional relationship of 

25 the lane marker between frames. Further, only when it is judged 
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1 that the lane marker is suitable, the value of the vanishing point 

2 parallax DP or the parallax correction value is changed. Hence, 

3 since an inappropriate vanishing point JV2D can be prevented from 

4 being calculated, this providing further stable, highly reliable 

5 distance information. 

6 In the above description, the updating of the vanishing 

7 point parallax is performed by the proportional control , however, 

8 the updating may be performed by the statistic control. For 

9 example, preparing a histogram composed of 1000 samples of the 

10 deviation amount A DP of the vanishing point parallax, a most 

11 frequently observed value may be used as a deviation amount A 

12 DP. This up-dating process according to the statistical control 

13 can be can be applied to a second, third, and fourth embodiments. 

14 (Second embodiment) 

15 According to the second embodiment, the parallax 

16 correction value DP is updated based on the comparison 

17 relationship between the gradient a of the actual road surface 

18 height Lr (that is, gradient a of the vanishing line Lv) and the 

19 gradient a' (that is, the parameter C identified from the lane 

20 marker models L, R) of the calculated road surface height Lr' . 

21 The steps of up-dating are the same as those shown in the 

22 flowcharts of Figs. 2 and 3. A portion different from the first 

23 embodiment is the step 11 of Fig. 3, that is, a part where the 

24 distance calculation parameter is updated. 

25 Fig. 5 is a flowchart showing up-dating steps of the 
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1 parallax correction value DP according to the second embodiment. 

2 First, at a step 31, it is judged whether or not the product of 

3 subtracting the gradient a of the actual road surface height Lr 

4 from the gradient a' of the calculated road surface height Lr' , 

5 is larger than a positive threshold value TH . In case where the 

6 positive judgment (YES) is made at the step 31, the program goes 

7 to a step 34 where a specified value a is added to the present 

8 value of the vanishing point parallax DP and at a step 36 a larger 

9 vanishing point parallax DP than a previous one is outputted to 

10 the recognition section 10. On the other hand, in case of NO at 

11 the step 31, the program goes to a step 32. 

12 At the step 32, it is judged whether or not the 

13 subtraction of the gradient a from the gradient a' is smaller 

14 than a negative threshold value -TH . In case of Yes at the step 

15 32, at a step 34, the specified CC is reduced from the present 

16 value of the vanishing point parallax DP. Accordingly, at a step 

17 36, a smaller vanishing point parallax DP than the previous one 

18 is outputted to the recognition section 10. On the other hand, 

19 in case of NO at the step 32, that is, in case where the subtraction 

20 a' -a is within a range from the negative threshold value -TH to 

21 the positive threshold value TH, the value DP is not changed based 

22 on the judgment that the vanishing point parallax DP is proper 

23 to maintain the control stability. 

24 The relationship between the difference of the 

25 gradient a' of the calculated road surface height Lr' from the 
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1 gradient a of the actual road surface height Lr and the distance 

2 Z, will be described by reference to Fig. 10. 

3 Letting the distance to a road surface point PI be zl, 

4 and letting the gradient of the actual road surface height Lr 

5 passing through the road surface point PI be a, when the distance 

6 zl' (containing an error) is calculated, a road surface point 

7 PI' on Z-X plane appears on a line m connecting the installation 

8 height of the camera CAH and the original road surface point PI. 

9 Accordingly, it is understood that as the calculated distance 

10 zl' becomes smaller than the actual distance zl, the gradient 

11 a' of the calculated road surface height Lr' becomes larger than 

12 the gradient a of the actual road surface height Lr. From this 

13 point of view, in case of a' >a, the calculated distance zl' should 

14 be adjusted so as to increase and for that purpose the value of 

15 the vanishing point parallax DP should be increased (see the 

16 formula 2) . Inversely, in case of a'<a, the calculated distance 

17 zl' should be adjusted to become small and for this purpose the 

18 value of the vanishing point parallax DP should be decreased. 

19 Even in case where the vanishing point parallax DP 

20 is not proper, that value gradually comes close to the proper 

21 value by carrying out the aforesaid flowchart in respective cycles . 

22 Hence, since the error of the distance Z caused by the horizontal 

23 deviation of the stereoscopic camera is gradually offset by the 

24 vanishing point parallax DP, the gradient a' "of the calculated 

25 road surface height Lr' converges to the gradient a of the actual 
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1 road surface height Lr. As a result, also in this embodiment, 

2 a highly accurate distance can be obtained stably. Further, as 

3 a result of performing the monitoring control based on thus 

4 obtained distance, the reliability of the vehicle surroundings 

5 monitoring can be enhanced. 

6 (Third embodiment) 

7 The feature of this embodiment is that an affine 

8 parameter SHFT1 (shift in horizontal direction) in the affine 

9 transformation is updated according to the difference between 

10 the gradient a' of the calculated road surface height Lr' and 

11 the gradient a of the actual road surface height Lr. 

12 Fig. 6 is a block diagram showing the construction of 

13 a stereoscopic type vehicle surroundings monitoring apparatus 

14 according to the third embodiment. The block diagram is the same 

15 as that of Fig. 1 except for that the affine parameter SHFT1 

16 calculated in the correction calculating section 13 is fed back 

17 to the correction circuit 5. Therefore, the components of the 

18 block diagram which are identical in both embodiments are denoted 

19 by identical reference numbers and are not described in detail. 

20 The steps of updating the affine parameter SHFT1 are 

21 the same as the flowcharts shown in Figs. 2 and 3 in the first 

22 embodiment. What differs from the first embodiment is a step 11 

23 of Fig. 3 concerning the updating of parameters for calculating 

24 the distance. 

25 Fig. 7 is a flowchart showing steps for up-dating an 
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1 affine parameter SHFT1 (parallax correction value) which 

2 represents the shift in the horizontal direction. First, at a 

3 step 41, it is judged whether or not the product of subtracting 

4 the gradient a of the actual road surface height Lr from the 

5 gradient a' of the calculated road surface height Lr' , is larger 

6 than a positive threshold value TH . In case where the positive 

7 judgment (YES) is made at the step 41, the program goes to a step 

8 44 where a specified value [3 is subtracted from the present value 

9 of the affine parameter SHFT1 and at a step 46 a smaller affine 

10 parameter SHFT1 than a previous one is outputted to the correction 

11 circuit 5. On the other hand, in case of NO at the step 41, the 

12 program goes to a step 42. 

13 At the step 42, it is judged whether or not the 

14 subtraction a' -a is smaller than a negative threshold value -TH. 

15 If the judgment is YES at the step 42, the specified value (3 is 

16 added to the present value of the affine parameter SHFT1 at a 

17 step 45 and a larger affine parameter SHFT1 than a previous one 

18 is outputted to the correction circuit 5 (step 46) . On the other 

19 hand, if the judgment is NO at the step 42, that is, if the 

20 subtraction a' -a is within a range from the negative threshold 

21 value -TH to the positive threshold value TH, it is judged that 

22 the affine parameter SHFT1 is proper to maintain the control 

23 stability and this value is not changed. 

24 As described in the second embodiment, in case of a' >a, 

25 the calculated distance zl' should be adjusted so as to increase, 
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1 in other words , the parallax d should be reduced . For that purpose, 

2 the value of the affine parameter SHFTl should be established 

3 to be smaller than the previous one. That is, the affine parameter 

4 SHFTl is updated such that the shift amount in the horizontal 

5 direction becomes small. Inversely, in case of a'<a, the 

6 calculated distance zl' should be adjusted to become small, in 

7 other words , the parallax d should be increased . For this purpose, 

8 the value of the affine parameter SHFTl should be established 

9 to be larger than the previous one. That is, the affine parameter 

10 SHFTl is up-dated such that the shift amount in the horizontal 

11 direction becomes large. 

12 As described before, the feeback adjustment of the 

13 affine parameter SHFTl (representing the shift in the horizontal 

14 direction) is made in parallel with the monitoring control. As 

15 a result, even in case where the horizontal deviation of the 

16 stereoscopic camera occurs, the affect of the deviation is offset 

17 by the affine parameter SHFTl, thereby an accurate parallax d 

18 can be obtained. As a result , highly accurate distance information 

19 can be obtained, whereby the reliability of the vehicle 

20 surroundings monitoring can be enhanced. 

21 (Fourth embodiment) 

22 This embodiment relates to the method of regulating 

23 the established vanishing point V (IV, JV) used in the 

24 transformation formulas 3 and 4 for calculating coordinates (X, 

25 Y} showing the position of an object by utilizing the vanishing 
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1 point V2d (IV2D, JV2D) which is shown in Fig. 13. 

2 Fig. 14 is a block diagram showing a stereoscopic type 

3 vehicle surroundings monitoring apparatus according to a fourth 

4 embodiment. In the correction calculating section 13, the 

5 established vanishing point V(IV, JV) is updated based on the 

6 vanishing point V2d(IV2D / JV2D) in the reference image and the 

7 calculated vanishing point IV, JV is outputted to the recognition 

8 section 10. Except for this section, the block diagram is 

9 identical to that of Fig. 1. Therefore, identical reference 

10 numbers denoted 

11 in both embodiments are not described in detail. 

12 Next, steps for updating the established vanishing 

13 point IV, JV will be described. First, according to the steps 

14 from the step 1 to the step 6 shown in the flowchart of Fig. 2, 

15 it is judged whether or not the reference image is in a condition 

16 suitable for calculating the vanishing point J2d (IV2D, JV2D) . 

17 Fig. 15 is a flowchart according to this embodiment 

18 continued from Fig. 2 and related to the updating process of the 

19 established vanishing point V (IV, JV) . First, at a step 51, 

20 an approximation line LI of a plurality of left white line edges 

21 Pedgel existing within a specified distance range (for example, 

22 0 to Z2) is calculated by the least square method (see Fig. 13) . 

23 Also, in the same manner, at the step 51, an approximation line 

24 L2 of a plurality of right white line edges Pedge2 existing within 

25 the distance range is calculated by the least square method. After 
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1 that, the program goes to a step 52 where a point of intersection 

2 of both approximation lines LI, L2, that is, a vanishing point 

3 J2d (IV2D, JV2D) of the reference image is calculated. 

4 At a step 53 following the step 52, the established 

5 vanishing point V (IV, JV) which is employed in the formulas 3 

6 and 4, is updated. First, the present value of an i coordinate 

7 value IV of the established vanishing point V is compared with 

8 an i coordinate value IV2D calculated at the step 52 and based 

9 on the result of the comparison, the vanishing point IV is updated 

10 by the following proportional control: 

11 [Updating of vanishing point IV] 

12 In case of IV - IV2D > TH IV <- IV -y 

13 In case of IV - IV2D < - TH IV <- IV + J 

14 In case of | IV -IV2D | ^ TH IV <- IV 

15 where 7 is a constant (0 <7<1). 

16 That is, in case where the established vanishing point 

17 IV is larger than the vanishing point IV2D identified from the 

18 left and right lane markers in the image, this case means that 

19 the established vanishing point IV deviates rightward in the 

20 horizontal direction of the image. In this case, the established 

21 vanishing point IV is shifted leftward by a specified amount by 

22 subtracting the constant y from the present value of the 

23 established vanishing point IV. On the other hand, in case where 

24 the established vanishing point IV is smaller than the vanishing 

25 point IV2D, this case means that the established vanishing point 
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1 IV deviates leftward in the horizontal direction of the image. 

2 In this case, the established vanishing point IV is shifted 

3 rightward by a specified amount by adding the constant 7 to the 

4 present value of the established vanishing point IV. Further, 

5 in order to make the control stable, in case where the difference 

6 (absolute value) between both is within a specified value TH, 

7 the established vanishing point IV is not changed. 

8 Similarly, the vanishing point JV is updated according 

9 to the following proportional control by comparing the present 

10 value of the j coordinate value JV of the established vanishing 

11 point V with the j coordinate value JV2D of the calculated 

12 vanishing point V2d. 

13 [Updating of vanishing point JV] 

14 In case of JV - JV2D > TH JV JV - 6 

15 In case of JV - JV2D < - TH JV <~ JV + <5 

16 In case of | JV -JV2D | ^ TH JV <- JV 

17 where(5is a constant (0 <(5<1) . 

18 That is, in case where the established vanishing point 

19 JV is larger than the vanishing point JV2D identified from the 

20 left and right lane markers in the image, this case means that 

21 the established vanishing point JV deviates upward in the vertical 

22 direction of the image. In this case, the established vanishing 

23 point JV is shifted downward by a specified amount by subtracting 

24 the constant 6 from the present value of the established 

25 vanishing point JV. On the other hand, in case where the 
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1 established vanishing point JV is smaller than the vanishing point 

2 JV2D, this case means that the established vanishing point JV 

3 deviates downward in the vertical direction of the image. In this 

4 case, the established vanishing point JV is shifted upward by 

5 a specified amount by adding the constant 6 to the present value 

6 of the established vanishing point JV. Further, in order to make 

7 the control stable, in case where the difference (absolute value) 

8 between both is within a specified value TH, the established 

9 vanishing point JV is not changed. 

10 At a step 54 following the step 53, the vanishing point 

11 V (IV, JV) is outputted to the recognition section 10. 

12 When the established vanishing point (IV, JV) is not 

13 proper, that value gradually comes close to a proper value by 

14 carrying out the aforesaid flowchart in each cycle. Specifically, 

15 this flow of control is performed in real time in parallel with 

16 the normal monitoring control and even when errors are caused 

17 in the present value of the established vanishing point (IV, JV) , 

18 that value containing errors gradually converges to an optimum 

19 value. As a result, the position (X, Y) of an object can be 

20 calculated with high precision, thereby the reliability of 

21 vehicle surroundings monitoring can be enhanced. 

22 [Application to miscellaneous monitoring apparatuses] 

23 In the embodiments described before, the method of 

24 calculating the vanishing point using the left and right lane 

25 markers projected on the image has been explained. This method 
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1 is based on a general tendency that, in case of monitoring ahead 

2 of the vehicle, there exist lane markers extending in the front 

3 (depth) direction of the vehicle on left and right sides of the 

4 road and these lane markers are parallel with each other. In the 

5 specification, a linear object like lane markers which extend 

6 in the front direction in parallel with each other, and which 

7 is a base for calculating an vanishing point, is referred to as 

8 "reference object" . The present invention can be broadly applied 

9 to miscellaneous monitoring system using picture images where 

10 the "reference object" is projected. 

11 Taking an example, in case of applying to an indoor 

12 robot able to recognize surrounding situations, a boundary line 

13 constituted by a wall and a floor can be used as a "reference 

14 object". Fig. 16 is an example of an image taken by an indoor 

15 robot. Normally, in many cases, the boundary line of a left wall 

16 and a floor and the boundary line of a right wall and a floor 

17 extend in the depth direction of the image in parallel with each 

18 other. Accordingly, the correction of the vanishing point or the 

19 correction of distance can be done by using the left and right 

20 boundary lines. 

21 Below, the outline of steps for adjusting the vanishing 

22 point making use of boundary lines. 

23 First, a plurality of lines LI, L2 are detected based 

24 on the reference image. In the same way as the condition of white 

25 line edges described before, conditions with respect to 
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1 brightness edges or parallax at the boundary portion between wall 

2 and floor are established before hand. Further, portions 

3 satisfying these conditions are recognized as boundary lines in 

4 the image and the linearity of these boundary lines is evaluated. 

5 After these processes, approximation lines LI, L2 are calculated . 

6 In another way, lines LI, L2 as "reference object" may be 

7 calculated by extracting dots (edge pixels at boundary portions) 

8 for forming lines in the image, using well-known Huff 

9 transformation and the like. 

10 Next, it is judged that the lines LI, L2 are 

11 approximately parallel with each other based on the distance image . 

12 As described before, the position of respective areas 

13 constituting lines LI, L2 in real space can be identified based 

14 on the distance image. Accordingly, in case where two lines LI, 

15 L2 are detected, the parallelism of these lines LI, L2 is judged 

16 using the known method. 

17 In case where the lines LI, L2 are parallel, a vanishing 

18 point is calculated from the point of intersection of these lines 

19 LI, L2 . Further, a gradient a of lines LI, L2 is calculated 

20 respectively and coordinates of the vanishing point are 

21 calculated based on the gradient. Finally, the value of the 

22 vanishing point parallax is adjusted such that the coordinates 

23 of two calculated vanishing points agree with each other. 

24 Further, taking another example, in case of applying 

25 to the system for monitoring frontal situations of a railway 
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1 rolling stock, left and right railways can be utilized as 

2 "reference object" . Fig. 17 is an example of the image projecting 

3 the front scenery of the railway rolling stock. The left and right 

4 railways extend in the depth direction in parallel with each other. 

5 Accordingly, two parallel lines LI, L2 can be identified by making 

6 use of the left and right railways as "reference object", thereby 

7 the vanishing points can be adjusted by the method described 

8 above . 

9 In summary, according to the present invention, 

10 parameters with respect to the calculation of three-dimensional 

11 information such as distance information, for example, a 

12 vanishing point parallax DP, an affine parameter SHFTI, a 

13 vanishing point (IV, JV) and the like, are corrected based on 

14 the actual vanishing point calculated from the left and right 

15 lane markers in the image . Accordingly, in case where a positional 

16 deviation of the stereoscopic camera occurs, since the parameters 

17 values are automatically adjusted so as to offset errors caused 

18 by that positional deviation, three-dimensional information (for 

19 example, distance information) with high accuracy can be obtained 

20 stably. 

21 While the presently preferred embodiments of the 

22 present invention have been shown and described, it is to be 

23 understood that these disclosures are for the purpose of 

24 illustration and that various changes and modifications may be 

25 made without departing from the scope of the invention as set 



41 



1 forth in the appended claims. 
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1 WHAT IS CLAIMED IS: 

2 1. A distance correcting apparatus of a surroundings 

3 monitoring system, comprising: 

4 a stereo imaging means for stereoscopically taking a 

5 pair of images; 

6 a parallax calculating means for calculating a 

7 parallax based on said pair of images; 

8 a distance calculating means for calculating a 

9 distance to an object based on said parallax and a first parameter 

10 for correcting said distance; 

11 an approximation line calculating means for 

12 calculating a plurality of approximation lines extending in the 

13 distance direction in parallel with each other based on said 

14 images; 

15 a vanishing point calculating means for calculating 

16 a vanishing point of said images from a point of intersection 

17 of said approximation lines; and 

18 a parameter correcting means for correcting said 

19 first parameter based on said vanishing point. 
20 

21 2. The apparatus according to claim 1, further 

22 comprising: 

23 a reference object detecting means for detecting a 

24 plurality of reference objects extending in the distance 

25 direction in parallel with each other from a scenery projected 
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1 in said images and for identifying a position of said reference 

2 objects in an image plane of said images. 
3 

4 3. The apparatus according to claim 2, wherein 

5 said vanishing point calculating means calculates an 

6 approximation line in said image plane for respective reference 

7 objects, when a plurality of reference objects are detected by 

8 said reference objects detecting means. 
9 

10 4. The apparatus according to claim 2, wherein 

11 said reference objects are lane markers on a road 

12 projected in said images and when left and right lane markers 

13 are detected on said road, said vanishing point calculating means 

14 calculates an approximation line in said image plane for said 

15 respective left and right lane markers. 
16 

17 5. The apparatus according to claim 4, wherein 

18 said vanishing point calculating means calculates said 

19 approximation line based on said left and right lane markers 

20 existing within a specified distance range. 
21 

22 6. The apparatus according to claim 4, wherein 

23 said reference object detecting means calculates a 

24 lane marker model expressing the change of a road surface height 

25 with respect to distance and said first parameter correcting means 
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1 identifies a condition of change of an actual road surface height 

2 based on said vanishing point calculated by said vanishing point 

3 calculating means, identifies a condition of change of a 

4 calculated road surface height based on said lane marker model 

5 calculated by said reference obj ect detecting means , and corrects 

6 said first parameter so that said condition of change of said 

7 calculated road surface height comes close to said condition of 

8 change of said actual road surface height. 
9 

10 7. The apparatus according to claim 4, wherein 

11 said reference object detecting means calculates a 

12 lane marker model expressing the change of a road surface height 

13 with respect to distance and said parameter correcting means 

14 identifies a first gradient indicating the change of a road 

15 surface height with respect to distance based on said vanishing 

16 point calculated by said vanishing point calculating means, 

17 identifies a second gradient indicating the change of a road 

18 surface height with respect to distance based on said lane marker 

19 model calculated by said reference object detecting means, and 

20 corrects said first parameter so that a deviation of said second 

21 gradient with respect to said first gradient becomes small. 
22 

23 8. The apparatus according to claim 4, wherein 

24 said vanishing point calculating means judges whether 

25 or not a lane marker projected in said images is a straight line 
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1 and in case where it is judged that said lane marker is a straight 

2 line, calculates said vanishing point of said images. 
3 

4 9. The apparatus according to claim 8, wherein 

5 said vanishing point calculating means evaluates a 

6 time-versus change of the position of a lane marker projected 

7 in said images, if said time-versus change is small, judges that 

8 said lane marker has a high reliability as lane markers, and 

9 calculates said vanishing point in said images. 
10 

11 10. The apparatus according to claim 9, wherein 

12 said parameter is a vanishing point parallax. 
13 

14 11. A distance correcting apparatus of a surroundings 

15 monitoring system, comprising: 

16 a stereo imaging means for stereoscopically taking a 

17 pair of images; 

18 a transforming means for geometrically transforming 

19 said pair of images based on a second parameter indicating a 

20 transference in the horizontal direction; 

21 a parallax calculating means for calculating a 

22 parallax based on said pair of images outputted from said 

23 transforming means; 

24 a distance calculating means for calculating a 

25 distance to an object based on said parallax; 
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1 a vanishing point calculating means for calculating 

2 a plurality of approximation lines extending in the distance 

3 direction in parallel with each other and calculating a vanishing 

4 point of said images from a point of intersection of said 

5 approximation lines; and 

6 a parameter correcting means for correcting said 

7 second parameter based on said vanishing point. 
8 

9 12. The apparatus according to claim 11, further 

10 comprising: 

11 a reference object detecting means for detecting a 

12 plurality of reference objects extending in the distance 

13 direction in parallel with each other from a scenery projected 

14 in said images and for identifying a position of said reference 

15 objects in an image plane of said images. 
16 

17 13. The apparatus according to claim 12, wherein 

18 said vanishing point calculating means calculates an 

19 approximation line in said image plane for respective reference 

20 objects, when a plurality of reference objects are detected by 

21 said reference objects detecting means. 
22 

23 14. The apparatus according to claim 12, wherein 

24 said reference objects are lane markers on a road 

25 projected in said images and when left and right lane markers 
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1 are detected on said road, said vanishing point calculating means 

2 calculates an approximation line in said image plane for said 

3 respective left and right lane markers. 
4 

5 is. The apparatus according to claim 14, wherein 

6 said vanishing point calculating means calculates said 

7 approximation line based on said left and right lane markers 

8 existing within a specified distance range. 
9 

10 16. The apparatus according to claim 14, wherein 

11 said reference object detecting means calculates a 

12 lane 

13 marker model expressing the change of a road surface height with 

14 respect to distance and said first parameter correcting means 

15 identifies a condition of change of an actual road surface height 

16 based on said vanishing point calculated by said vanishing point 

17 calculating means, identifies a condition of change of a 

18 calculated road surface height based on said lane marker model 

19 calculated by said reference object detecting means, and corrects 

20 said first parameter so that said condition of change of said 

21 calculated road surface height comes close to said condition of 

22 change of said actual road surface height. 
23 

24 17. The apparatus according to claim' 14, wherein 

25 said reference object detecting means calculates a 



48 



1 lane marker model expressing the change of a road surface height 

2 with respect to distance and said parameter correcting means 

3 identifies a third gradient indicating the change of a road 

4 surface height with respect to distance based on said vanishing 

5 point calculated by said vanishing point calculating means, 

6 identifies a fourth gradient indicating the change of a road 

7 surface height with respect to distance based on said lane marker 

8 model calculated by said reference object detecting means, and 

9 corrects said third parameter so that a deviation of said fourth 
10 gradient with respect to said third gradient becomes small. 
11 

12 18. The apparatus according to claim 14, wherein 

13 said vanishing point calculating means judges whether 

14 or not a lane marker projected in said images is a straight line 

15 and in case where it is judged that said lane marker is a straight 

16 line, calculates said vanishing point of said images. 
17 

18 19. The apparatus according to claim 18, wherein 

19 said vanishing point calculating means evaluates a 

20 time-versus change of the position of a lane marker projected 

21 in said images, if said time-versus change is small, judges that 

22 said lane marker has a high reliability as lane markers, and 

23 calculates said vanishing point in said images. 
24 

25 20. A vanishing point correcting apparatus of a surroundings 
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1 monitoring system for taking images of a scenery in front of an 

2 own vehicle and for obtaining a three-dimensional information 

3 of an object projected in said images by making use of an 

4 established vanishing point established beforehand, comprising: 

5 reference object detecting means for detecting lane 

6 markers on a road projected in said images and for identifying 

7 a position of said lane markers on an image plane of said images; 

8 vanishing point calculating means, when a left and 

9 right lane marker is detected on said road and it is judged that 

10 said lane marker projected in said images is a straight line, 

11 for calculating an approximation line in said image plane for 

12 said respective left and right lane markers and for calculating 

13 a vanishing point from a point of intersection of said 

14 approximation lines; and 

15 a vanishing point correcting means for correcting said 

16 vanishing point so that said established vanishing point comes 

17 close to said vanishing point calculated by said vanishing point 

18 calculating means. 
19 

20 21. The apparatus according to claim 20, wherein 

21 said vanishing point calculating means evaluates a 

22 time-versus change of the position of a lane marker projected 

23 in said images, if said time-versus change is small, judges that 

24 said lane marker has a high reliability as lane markers, and 

25 calculates said vanishing point in said images. 
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1 ABSTRACT 

2 A distance correcting apparatus of a surroundings 

3 monitoring system includes a stereo imaging means for 

4 stereoscopically taking a pair of images of a frontal scenery, 

5 a parallax calculating means for calculating a parallax based 

6 on the pair of images, a distance calculating means for 

7 calculating a distance to an object based on the parallax and 

8 a parameter for correcting distance, an approximation line 

9 calculating means for calculating a plurality of approximation 

10 lines extending in the distance direction in parallel with each 

11 other based on the images, a vanishing point calculating means 

12 for calculating a vanishing point of the images from a point of 

13 intersection of the approximation lines and a parameter 

14 correcting means for correcting the parameter based on the 

15 calculated vanishing point. 
16 
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